Information collages are physical artifacts that are assembled, manipulated, and maintained to create, alter, preserve, share, or coordinate information. For example, wall mounted pin boards or magnetic boards can be used by an individual or group to transiently or semi-permanently display documents, calendars, task schedules, phone number lists, project proposals, informational flyers, meeting announcements, photographs, maps, or any other desired information. In some cases, small physical artifacts such as keys or magnetic icons can even be attached to the surface.
Advantageously, such information collages allow for ready manipulation of surface mounted documents or artifacts through addition, removal, replacement, reordering, or stacking. For example, a team's project schedule might be represented on a wall mounted pin board by pin attached task cards arranged in a timeline format. Such task cards can be easily altered or physically moved with respect to a displayed timeline as people leave a project, schedules slip, or any other problems develop. Unfortunately, this easy alteration of information in the collage by individuals or group members presents its own set of problems. Attached task cards can be inappropriately removed, misplaced, or obscured. Recording state of the pin board at a given time for electronic archival or tracking purposes can require substantial efforts, and it is difficult to synchronize and transfer the information in the collage to group members not physically present at a site.
Accordingly, the present invention automatically or semi-automatically integrates a computational system with an information collage to preserve the advantages of conventional information collages while alleviating many problems associated with such collages. A physical information collage (e.g. a pin board with spatially differentiated pin mounted task cards, each card having a detectable identity tag) can be made part of a computational system by sensing the identity, position, and contents of the component artifacts. In effect, each task card has its content and position stored in a database, with movement of a particular task card automatically resulting in update of the database to record the new position of that task card.
In certain embodiments, an enhanced information collage has a surface that supports a plurality of detachable collage components. These collage components are typically documents, but may be physical icons, or other small physical artifacts attachable by magnets, pins, adhesives, frictional forces, or other conventional attachment mechanisms. An identification unit determines informational content of at least some of the plurality of collage components using various sensing modalities (including but not limited to camera, radio, infrared, radio, acoustic, or electrical based techniques). Information can be directly determined by transfer/reading of information (which may be either an address to an electronic document, or the document information itself) from a collage component, or determined by position based associations maintained between an electronic document and a collage component.
Position of the document can be determined by a location unit for tracking position of at least some of the plurality of collage components. Like the identification unit, the location unit can use various sensing modalities alone or in combination (including, but not limited to camera, radio, infrared, radio, acoustic, or electrical techniques) to track position of collage components. The identification unit and location unit together provides an information maintenance and archival system with information necessary for constructing a series of time dependent snapshots of position and informational content of at least some of the plurality of collage components. This information maintenance system is updateable to reflect changes in position and informational content of at least some of the plurality of collage components, while still retaining a time/action addressable record of state changes of the plurality of collage components. This allows, for example, a user to digitally replay changes in a collage, or determine the collage state a day, a week, or a month earlier.
As will be appreciated by those skilled in the art, certain collage components may not be merely associated with electronic data (e.g. an electronically available text document or picture), but may actually "operate" on other collage components to define data relationships between collage components. Data can be altered, combined, erased, saved, augmented, transferred, or otherwise modified through provision of various digital services associated with "operator icons". In operation, a collage component having associated electronic data is physically associated with an operator icon, causing modification of that associated electronic data in response to physical association of the operator icon with collage component. For example, consider a first collage component representing a document and a second collage component representing user of the information collage. An operator icon symbolically configured as a pointing arrow can be positioned on the surface of the information collage to point from the first collage component to the second collage component. This association causes the system to electronically mail the first document to the user. Reversal of the arrow so that the arrow points from the user to the document can result in modification of document properties so that the user is linked to the document and informed of any updates, changes, or alterations to the document. Alternatively, an arrow shaped operator icon can be used to link a document to a printer or other external digital service, depending on the particular operation or digital service previously registered with the information maintenance and archival system.
As those skilled in the art will appreciate, registration of operator properties can be dynamically defined by a user interacting with collage components. For example, a user can dynamically define a digital service associable with an operator icon by physically associating at least one additional operator icon with the operator icon; by associating user definable text with the operator icon; by associating audibly announced actions with the operator icon; by associating gestural actions with the operator icon; by associating a spatially discrete zone of the information collage with the operator icon, with the spatially discrete zone having an associated digital service (e.g. print, group, save), or by any other suitable user definable action interpretable by the system as an instruction to link a digital service to a generic operator icon. After the particular digital service linked to a digital service has been transiently, semi-permanently, or permanently defined by a user, the physical association between the operator icon and the collage component is used to determine the particular required data modifications (which may include digital services or other data parametrizable functions).
For example, in one possible embodiment of the present invention a generic bar shaped physical icon can be used as an audio level control for a sound system connected to the information collage. A user registers linkage of the bar shaped physical icon to the audio level control system by writing "VOLUME" next to the physical icon. These characters can be automatically recognized by a camera based character recognition and analysis system viewing the information collage, and necessary modifications made to registry databases in the information maintenance system. The actual level adjustments can be made by moving another physical icon along the bar shaped physical icon, an action that is interpreted as changing audio levels.
Advantageously, the foregoing system allows applications such as workflow organizers, spreadsheets, in/out boards, geographical information systems, outliners, software construction kits, software verification systems, or any other general or special-purpose software application (with or without connection to external digitally controlled or monitorable physical systems) to execute operations based on some interpretation of user activities ("events") on the information collage. These events can be interpreted implicitly; through use of explicit operator icons; through generic, user definable, operator icons; or as part of a "visual language".
As will be appreciated, any sensed user action at the surface of an information collage can implicitly evoke an operation by an application. For example, putting a person icon next to a task card can assign the task to the person and email a reminder to the person. This is accomplished by a special-purpose application that knows how to interpret actions on the particular information collage, which in this case is used for group task management. The implicit actions on the board of the information collage will generally require the system to recognize and interpret spatial relationships (e.g., adjacency, alignment, enclosure) among collage components, diagrammatic annotations (e.g., encirclements or links), manual gestures, or other modes of interactions with the collage.
In contrast, operations can be explicitly invoked through operator icons that direct data flow or otherwise initiate invocation of application mediated services. For example, an operator icon can be associated with a particular operation (e.g., print). Actions on or with that icon can cause particular operations (i.e. the operator on specific operands) to be invoked. The actions may involve placing the operator icon in various spatial relationships, binding the operator to certain actions or objects (e.g., a link drawn between it and an operand), or acting in a temporal sequence (e.g., touching an operator icon and then touching another icon on which the operation is performed), etc.
Explicit operator icons can be defined in advance of use (e.g., a printer shaped icon that "prints" documents to which it is touched). Alternatively, operator icons can be dynamically defined by a user of an information collage (e.g., a square icon on which a user first writes the word "print" thereafter causes printing of documents to which it is touched). Sets of icons can be grouped together by touching, stacking, aligning, etc, to form user-defined "packages" of icons that can be to define various particular actions (e.g. using stacked icons for printing and saving to file for invoking an application to print and save to file a document touched by the compound icon). In a broad sense, operator icons and other information collage components can define a visual language. For example, a user can construct an interpretable command ("imperative sentence") through spatially or temporally arranged groups of icons.
Additional functions, objects, advantages, and features of the present invention will become apparent from consideration of the following description and drawings of preferred embodiments.